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ABSTRACT 

The role of educational evaluation in product 
development is part of a framework for discussing and organizing 
formative evaluation studies. This framework is two dimensional, with 
formative evaluation activity as one dimension and sources of 
information as the other dimension. Discussions are presented on the 
four categories of formative evaluation activities — predevelopmental, 
evaluation of objectives, formative interim evaluation, and formative 
product evaluation — and on the three major components of sources of 
Information — internal, external, and concextual— and types of 
technique most relevant to a particular category. It is recommended 
that the activities described become an integral part of any 
systematic development effort to the extent that project resources 
and constraints allow. Three figures present the following: A 
Structure for Planning Formative Evaluation in Product Development; A 
Classification of Information Needs in Formative Evaluation; and 
Summary of Techniques and Procedures Appropriate for Formative 
Evaluation. (DB) 
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A Structure for Formative Evaluation in 
product Development 

James R. Sanders and Donald J. Cunningham 
Educational Research Institute for 

and Child Study 

Evaluation Laboratory 

Indiana University 

Many educator? working on the development of educational products have 
stated a need for a logical structure for formative evaluation functions since 
Scrlven (1967) made the distinction between formative and summatlve evaluation. 
Recent writers (viz., Reynolds and Light* 1971, Abedor, 1971, Tate, 1971 >, 

Westbury, 1970, and Weiss, 1971) have noted the ambiguity in definition of the 
term, formative evaluation, and the consequent paucity of well-defined procedures 
and tec hn iques for conducting such evaluation studies. The need for a framework 
for discussing and organizing formative evaluation studies has become evident. 

The most urgent need, judging from the frequency of pleas In the lierature, appears 
to be in the area of product development. The questions of how to systematically 
put limits on formative evaluation studies, and how to choose appropriate evaluation 
methods once the limits are set, appear to have been especially difficult to 
answer by developers. The purpose of this paper is to focus on the role of 
educational evaluation in product development In order to present a framework 
appropriate for discussing and organizing formative evaluation studies. The 
framework is a result of the synthesis of various development and evaluation metiels 
and a review of case histories in education development. 

The term evaluation will be used herein to refer to the process of determin- 
ing the value or worth of a process or product.. Formative evaluation will refer to 
the process of judging a fluid process or product that can be revised in form. The 
results of formative evaluation studies are gfren to persons directly involved in the 

process or developing the product. 
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The Framework 

Our current version of this framework is two dimensional with formative 
evaluation activity constituting one dimension and sources of information con- 
stituting the other. Firgure 1 represents the four stages of formative evaluation 
functions which we have delineated and labeled pre-developmental activities, 
evaluation of objectives activities, formative interim evaluation activities, and 
formative product evaluation activities. Figure 2 elaborates the second dimension 
and its three major components; internal, external, and contextual sources of 
information. Figure 3 shows these two dimensions conjoined in a summary table. 

The information dimesnion has already been discussed by the second author elsewhere 
(Cunningham, 1971) and will be only briefly summarized here. The discussion will 
center on the four categories of formative evaluation activities and on the sources 
of information and types of technique most relevant to a particular category. 

Sources, of Information 

Three major sources o£ formative evaluation Information concerning products 
can be Identified. The first has bean labeled by programed instruction enthusiasts 
(Lumsdaine, 1965) as Internal Information, or Information which can be generated 
by inspecting the product itself. Included in this category Is descriptive Informa- 
tion concerning the product: Its physical characteristics. Its content, etc. Also 

included in the internal category Is critical appraisal of the product or its 
components by specified persons such as students, teachers, subject matter experts, 
media experts, etc. Descriptive information and critical appraisals are alUce In 
that they are both generated by inspection but differ la that descriptive Informs- 

tion is often less controversial than critical appraisal. 

The second major source of information is external information defined 
here as information concerning the effects of the product or its components on the 
behaviors of students, teachers, parents, and other relevant groups. These effects 
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could include , for instance, student achievement from a programed text on genetics 
or the attitudes of parents towards the objectives of such a program before the 
product is actually developed* The distinction between critical appraisal and 
certain categories of external information is subtle but important* In the one 
case classes of people are used to evaluate the worth of the product using their 
own standards , while in the other case, the effects of exposure to this product 
on specified populations is ascertained using a public, across-the-board set of 
criteria. 

The third and last major source of information is contextual information 
and refers to information concerning the conditions under which the materials are 
expected to function* Product developers make frequent assumptions covering the 
context within which their product will function but only rarely test those 
assumptions. Examples of such information are pupil characteristics (entering 
behavior), teacher characteristics, curricular context, etc. 

These three sources of information become salient to a greater or lesser 
extent as we consider each of the four categories of formative evaluation activity 
specified in Figure 1* We now turn to a discussion of those stages with illus- 
trations of the information needs at each. 

Pre - Developmental Activities 

The first category is labeled as a pre-developmental set of activities since 
such work should occur before product development is even started. Manion (1971) 
has noted that a prerequisite for good development is the availability of reliable 
information regarding the needs of a given population* Although Stuff lebeam 
(1965, 1968, 1971) has labeled the activities in this category as evaluative 
activities (his term is context evaluation) , there is no clear worth-giving component 
to the assessment of needs and, thus, we prefer to avoid labeling needs assessment 
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as evaluation. Needs assessment, as a pre-developmental activity involves the 
collection of evidence that some situation in education isn*t as it should be. 

Popham (1970), has described needs assessment as identifying discrepancies between 
desired outcomes and current status. Inherent in that definition are difficulties 
with operationally defining desired outcomes and finding reliable, content valid 
measures of current status. Popham suggested using the CSE Instructional Objectives 
Exchange 1 (IOX) for lists of desired outcomes (instructional objectives) and items 
to assess the outcomes* When educators move outside the realm of traditional 
instruction, however, sources of goal statements become scarce or nonexistent. For 
instance, a program planner considering an open school program in the Pittsburgh 
Public Schools is going to be hard-pressed to find a list of objectives for open 
school programs. 

Exemplary applications of needs assessment procedures have been provided 
by Glass (1969) and Womer (1970). Briefly, the steps of a needs assessment design 
should include the following steps: 

1. Collect an exhaustive list of objectives from a wide range of sources. 

2. Have the objectives reviewed, revised, supplemented, and reduced 
by representatives of groups affected by the program. 

3. Prepare one or more test items for every objective in the final 
set. Where complex behaviors are being assessed, the assessor will 
certainly have to devise information collection methods other than 
paper and pencil techniques. 

Center for the Study of Evaluation, Instructional Objectives Exchange , Box 24095, 
Los Angeles, California 90024. 
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4# Prepare a sampling frame for collecting status information 
(see Cochran, 1963). 

5. Develop methods for data analysis and reporting of the information. 

The assessor will want to decide on the level of data summarization 
(l.e., by classroom, by school, by geographical region) and criteria 
for defining needs* Steps should also be taken to create an informa* 

tion storage and retrieval system, 

6. Train data collectors, prepare the assessment sites, prepare an 
assessment time schedule, and collect the information* 

The assessment procedure should not remain static, but should continually undergo 
revision as feedback about the psychometric characteristics of the information 
collection instruments, the sampling plan, and the adequacy of the set of objectives 
(values do change) is received. It is obvious that a systematic needs assessment 
will serve to delimit the parameters of subsequent product development efforts* 

Once a set of needs is identified, the setting of priorities among the 
needs is important* The evaluation of needs should be central to any educational 
development effort that lacks unlimited funding* The methods for the evaluation 
of needs should parallel those used for the evaluation of objectives. Thus, the 
methods described in the following section are equally applicable here. The 
appraisal of identified needs will lead to a priority ranking of need areas. 

Hence, there is some evaluative information that developers must attend to even 
before program objectives are considered. Scriven (1967) noted that the evalu- 
ation activity consists of gathering and combining of data with a weighted set 
of standards (desired outcomes in needs assessment) to yield numerical ratings. 

We are suggesting a similar procedure for evaluating needs whereby a formative 
evaluator should collect judgment data (e.g., Q-sort data) and, when conflicts 
are minimal, weight the needs accordingly. When conflicts arise further analysis 
of these specific conflicting needs is necessary. 
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The information needs in predevelopraental formative evaluation activities 
fit nicely into the three categories described earlier. Contextual information 
is generated concerning the present status of student behaviors, teacher competencies, 
school resources, and other relevant information. This information will be 
invaluable to the product developer in helping him conceptualize the intended 
context of his product. Most products are developed with a particular “average* 1 
student and teaching context in mind. A needs assessment can test the adequacy 
of these assumptions and whether the product is. really needed. 

External information is not, of course, directly involved in the predevelop- 
ment stage since the product whose effects arc being evaluated has not yet been 
developed. But in another sense, external information plays a key role at this 
stage. Criterion behaviors are being established, those which will be used, in 
part at least, to gauge the merit of the product. Certainly the objectives and 
measures used to index those objectives generated by a thorough needs assessment 
will transend any one product. But the initial conceptualization of objectives and 
the subsequent development of criterion— referenced measurement techniques (e.g. 
Glaser, 1963; Popham and Huselc, 1968; Womer, 1971) will have allowed the specif ication 
at least some of the external information that will be needed when a product is 
produced. One danger which should be avoided is too heavy a reliance on external 
information generated at the needs assessment stage. Other variables unique to 
the particular product must also be examined. 

Information which we have labeled internal information enters at this stage 
in at least two places: specifying expeditions for school programs and setting 

need priorities. The generation of a list of existing objectives for school programs 
is an especially difficult task. Such lists are often not available. Educators 
often feel much more comfortable with implicit rather than articulated goal state- 
ments. Stake (1970) has reviewed some of the techniques which can be used if no 
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set of objectives is available. Such methods as requesting interested groups or 
relevant experts to generate a list o£ expectations, analysing documents (e.g. 
textbooks, policy statements, reports of related projects, etc.) which might allow 
the evaluator to objectively determine those objectives, can have substantial payoff 
for this type of formative evaluation activity. 

Evaluation of Objectives 

At the beginning of the development process, the developer will undoubtedly 
go about the task of writing objectives for his specific product, at least to use 
as reminders of the direction in which he is going. Once the objectives have been 
prepared, the formative evaluator again has a task defined for him - the evaluation 
of the objectives. Scriven (1967) and many others have noted that this role is one 
of the most important evaluation roles in product development. Specifically, the 
developer ought to have some reliable information about the worth of his objectives 
to avoid investing his scarce resources in developing materials to bring about out- 
comes of questionable value. He is vulnerable to the possibility of producing a 
set of elaborate materials that are worthless only because his original objectives 

were inadequate. 

Stake (1970) has suggested two types of analysis that may be used to 
evaluate objectives (or needs, as indicated earlier) - logical and empirical. Both 
of these analyses would use internal information in the scheme advocated here. 
Although we in education have accumulated a large amount of knowledge about 
relevant empirical analytical procedures and techniques, we are just beginning to 

discover appropriate logical analytical methods* 

Logical Analyses . Clayton (undated) has Implied that certain methods drawn 
from the discipline of philosophy could be used to determine the validity of priority 
and value statements. In essence, an objective is a value or priority statement. 

It is a statement about a desired outcome. Some priority has been assigned to 
that outcome. One logical procedure for establishing the value of a set of 
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objectives is to examine the cogency of the argument or rational behind each 
objective. If there are no justifiable reasons for listing an objective, it 
cannot be too valuable. Here is one point where a needs assessment could facilitate 
matters greatly for the educational developer. If he has evidence on which to 
base his arguments for a set of objectives, few people are going to question the 
appropriateness of such objectives. 

A second method for logically analyzing an objective is to examine the 
consequences of accomplishing it. In many cases it will be hard to foresee 
consequences or discriminate among consequences of alternate objectives. However, 
when consequences can be predicted and alternate objectives are associated with 
discriminable consequences, then this method of analysis will have some pay-off. 

A search of the educational research literature will sometimes reveal the implica- 
tions of achieving specific objectives. This information should not be ignored 
when objectives are being evaluated. 

A third method for logically analyzing an objective is to appeal to higher- 
order value statements. If a program objective is in conflict with a policy 
statement or a state that everyone feels is desirable, then the objective should 

be reconsidered. 

Scriven (1966) suggested another, similar kind of val? a-reasoning. He 
maintained the following: 

1. If something will bring about a state of affairs that people value, 
that is a good prima facie reason for doing it. 

2. If there are prima facie reasons for doing something and none against, 

we should do it. 

3. If there is a conflict of supportable prima facie reasons, appeal 
must be made to a general moral principle. 
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E mpirical Analyses . When judging any important part of the educational pro- 
cess it is imp ortant to the existence of diverse value positions. Stake (1970) 

provided an excellent review of methods for the collection of evidence about how 
widespread a value position actually is. One category of data collection is the 
collection of group data. Stake listed four ways of obtaining such data: 

1. surveys 

2. scaling 

3. Q - tecnique 

4. semantic differential 

We might add the delphi technique (Helmer, 1967) , a special instance of survey 
methodology, as another useful evaluation tool. Archival files containing aggregate 
group judgments should also not be overlooked. An indirect measure of group judg- 
ment, such as the observation of groups behaviors under simulated conditions, might 
also provide useful information. For example, an observation of a school board f s 
response to a "what if I do ..." statement could provide the evaluator with 
meaningful judgment data. This type of measure could easily fall under Stake* s 
second category of data collection methods also. Instances of group analysis are 
frequently found in the literature, but few evaluators have recognized the utility 
of this methodology for the evaluation of program objectives or needs. Stratifying 
a population into ethic groups or types of community to collect judgment data is 
necessary if a high-impact educational product is to be planned or developed in 
the right direction. Manion (1971) described methods used in a regional laboratory 
to collect just this kind of data using techniques such as the Q-sort. Johnson 
(1971) mentioned the valua of using survey methods to collect judgment data. The 
only limitations to using diverse methods for collecting group judgment data is 
our lack of imagination. 
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A second category of empirical evidence suggested by Stake is data collected 
from observations and from the experts. Widespread collection of such Judgment 
data is essential if we are to achieve the objective, reliable, challengeable, 
open, due-process society that Campbell (1970) has called for. It is encouraging 
to see educators present their work for critical appraisal by independent judges. 
Methods suggested by Abedor (1971) are illustrative of the methods appropriate for 
the empirical analysis of objectives* Observation methods borrowed from sociology 
(e.g., the use of participant observers and other unobstrusive techniques, cf*, 

Webb et al . (1966) and Sechrist (1972)) and anthropology (e.g., the use of ethno- 
graphic techniques in the schools, cf. Smith and Geoffrey (I960) should be added to 
Stake's and Rosenshine's (1970) observation methods developed in psychology. This 
category of judgment data is a restatement of the professional judgment” paradigm 
currently used by accrediation agencies, in doctoral oral examinations, wad by 
federal proposal reading panels. Few instances of the use of a professional review 
panel to evaluate objectives or needs are found in the literature, however. The 
National Assessment of Educational Progress project and the assessment program in 
progress in the state of Colorado contain procedures worth noting. 

The third category of empirical analysis is the analysis of documents. 

There may be policy positions taken by a school district or by a community that 
have been filed away or that appear in a newspaper. State departments of education 
typically have impressive archives containing value statements made by relevant 
Individuals and groups. Weiss (1971) provided an argument for the use of content 
analysis in formative product evaluation studies. This technique is Just as useful, 
however, in collecting Judgment data for evaluating objectives. The content analysis 
of speeches made by the President of the United States since he took office would 
provide interesting data about the consistency of his attitude toward issues in 
public education. Changes in his values uncovered by a content analysis could great* 
ly affect the acceptability of certain objectives in a large development project. 
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The foregoing techniques of evaluating product objectives have bean largely 
internal in character; that is, the evaluation of objectives has bean accomplished 
by examining the objectives themselves* This is as it should be since the effects 
of exposure only to the objectives of a product is likely to be different than 
exposure to the product# Perhaps seme typos of external information collected at 
this stage could prove useful, however, since one measure of the worth of objectives 
could be the extent to which the objectives can be measured and whether external 
measures can bo developed* Any set of objectives, no matter how well received by 
experts, but which cannot be validly operationalized has limited value# Studies of 
the effects of goal accomplishment might also be used in appraising objectives* 

A similar argument can be made for contextual information* Contextual in* 
formation concerning objectives could reflect the adequacy with which the objectives 
met the contextual requirements established in a needs assessment* If no needs 
assessment was undertaken, data concerning the intended context could be collected 
at this point to establish the need for the product# 

Formative Interim Evaluation 

Once the developer has begun work on his product, the formative evaluator 
will undoubtedly be called on to function in the role of resident appraiser* We 
have categorized his evaluation activities on early product development efforts 
as formative interim evaluation activities since he will be working with the pieces 
of a product that is not yet fully assembled# The last category of activity in 
our structure, formative product evaluation, deals with the appraisal of a final 
product which can still be revised* 

It is , useful to distinguish between formal and informal interim evaluation 
activities to allow for flexibility in any formative evaluation design. Formal 
procedures refer to structured data collection and appraisal activities while 
Informal procedures are generally unstructured and often unplanned# Internal, 
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external , and contextual Information can be collected using either formal or 
informal methods with the choice depending on such factors as the generalieabilifcy 
of results desired, the degree of intervention possible, convenience, etc* We 
shall suggest some Informal procedures later in the discussion* 

Interim formative evaluation which relies on external information has been 
called Pay-Off evaluation by Scriven (1967) • Pay-off evaluation is without a doubt 
the most common type of evaluation activity in instructional development; indeed for 
some this type is the only "real" type of evaluation. She methods used to collect 
external information are well known and will not be elaborated in detail here. (©•& 
see Metfessel & Michael, 1969) One point that should be emphasized, however, is 
that the appropriateness of methods for collecting external information depends up- 
on the stage of development of the product. For example, early in the development 
of materials, a clinical approach to student debriefing such as that suggested by 
Markle (1970) would be appropriate. Later in the development process, the use of 

true experimental designs may be most appropriate* 

She determination and evaluation of the effects of pieces of the final 
product (e.g* the effect of programed materials on student retention in a small 
sample) in an interim pay-off evaluation study provides useful information to the 
producer. Abedor (1971) contributed excellent illustrations of interim pay-off 
evaluation activities* Goodwin and Sanders (1971) reported the results of small 
school bus studies to determine the effects of alternative instructional treatments 
to youngsters riding the bus. The results and raconanandations of the school bus 
studies provided the basis for choosing the pieces of a large-scale busing-learning 
program. The **primary M evaluation (evaluation of the effects) of a small segment 
of the final product can provide useful information in the early stages of 
development. In order to determine the effects of materials on human behaviors, 
experimental and quasi -experimental designs (Campbell and Stanley, 1966) and 
quantitative naturalistic observation techniques (Butler, Rice, and Wagstaff , 1963) 
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become essential tools of the formative evaluator (both Interim and product)* 

A second type of formal interim evaluation activity may be labeled interim 
intrinsic evaluation (following Scriven, 1967)* By this, we mean the evaluation of 
t rans nation a l or means- to -the-ends program characteristics (^secondary ©valuation)* 
Such evaluative activities as the analysis of the content of program components or 
the appraisal of instructional strategies would be examples of activities that fall 
into this category* This type of evaluation obviously relies on internal informs* 
tion, both descriptive and critical appraisal. Some categories of descriptive in- 
formation are listed in Figure 2. It should be noted that those categories overlap 
substantially with descriptive guidelines developed by Morrissett and Stevens (1968, 
1971) , Tyler and Klein (1968) , and Eash (1970) . The identification of the physical 
characteristics of the materials provides information about the nominal stimuli 
which may potentially affect human behaviors* Information about the rationale, 
goals, and objectives of the product contributes to an understanding of value 
positions taken by the developer or other involved persons. Obviously, content 
analysis techniques can contribute greatly agam for formative intrinsic evaluation 
where content refers to an objective representation of the substance of the interim 
pieces of the product. Hierarchical analyses of product components (cf., Gagnd , 
1970) would also be a useful tool for the interim intrinsic formative evaluator. 

Critical appraisal is information generated from interested parties in- 
specting the materials. Stake (1970) referred to such information as Judgment data 
suggested techniques for collecting the information* in interim intrinsic eval- 
uation work, the judgment data should be used strictly as feedback to the developer* 
The backgrounds a nd standards of the appraisers should be of concern to the developer 
only for providing information useful in weighting the feedback. The use of 
critical appra i se r s is quite common in media* The studies conducted by Koban &94$ , 
for instance, provided substantial information to film developers. One source of 
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critical appraisal is the author* himself* The author* under questioning, can 
provide valuable information about the materials he is producing. Walberg (1970) 
noted that "the first and most severe critic must be the developer. But his own 
criticism is not enough, for inevitably he will be biased and unable to see all the 
weak points of his work* Therefore* he must solicit critical opinion from his 
immediate colleagues and various outsiders * specialists in educational media and 
evaluation, university professors of the subject, and school teachers and students 
using trial versions of the course” • Little research has been done on variables 
affecting an author 1 s evaluation of his own materials in the early stages of de- 
velopment or planning. It would be interesting, for instance, to investigate the 
relationships between the author* s view of the learning process or his personality 
characteristics and the types of early revisions he makes. 

The term, experts* refers to persons trained in the subject matter* psycho- 
logical processes, philosophy, etc. covered by the plan or materials. Student* 
teacher* and other relevant audience feedback is important, too, in the early stages 
of development. As mentioned earlier. Stake (1970) has provided an excellent dis- 
cussion of the use of professionals in evaluation studies. The point to make, here 
is that interim intrinsic formative evaluators should also take advantage of the 
guidelines provided by Stake. 

The collection of contextual Information at the interim formative evaluation 
stage can not be accomplished in isolation from a particular set of objectives or 
a particular product since by definition the role of context is to sepcify the 
limits of the product. The task of the formative evaluator, therefore, is to 
establish whether predicted relationships between context, internal and external in- 
formation hold. Is it the case^ for instancy that students with specified entry be- 
haviors (context) learn more matehmatics (external) from a programmed text using hier- 
archial sequencing (internal)? At the interim formative evaluation stage, tryouts 
of the materials often include small groups of students who may not be completely 
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typical of the target population, under circumstances rather different from the 
intended curricular context. Thus, context variables are typically less well con- 
trolled at this stage* On the other hand, large scale field tests, in situations 
closely parallelling the intended context and conducted before sufficient interim 
evaluation has taken place, are likely to be very wasteful of time and effort. 
Context Information plays its greatest role in the last category of activity, 
formative product evaluation* 

A misconception we would like to correct is that evaluation procedures need 
always be planned and tightly structured. Indeed, the evaluator may often, through 
the use of informal (unstructured and often unplanned) methods, uncover many criti- 
cal dimensions of the developrent work* Webb, et al . (1966) and Sechrist (1972) 
have suggested types of information that might be collected as part of informal 
interim formative evaluation work* The following categories of nonreactive measures^ 
suggested by Webb, et al . (1966), should not be overlooked in formative interim 
evaluation work: 

1. Physical Traces. Wear and tear on first drafts of instructional 
material can be a reliable index of student and teacher appraisals of 
the material. The first author often circulates first drafts of pro- 
fessional papers to receive his colleagues comments. A recent circula- 
tion of two papers lead to one being returned without comment and 
completely unsoiled; the other was returned in terrible condition, with 
requests for copies from every reader* It*s not hard to infer which 
paper was the better product - at least in the eyes of this particular 
population* 

2. Public Archival Records* Newsletter content for a large curriculum 
project can be used, for example, for something other than the 
dissemination of information* The formative evaluator would be wise 
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to look for discrepancies between program plans and reported program 
activities in such documents# 

3# Private Records. The expense file, for instance, for a development 

project can produce startling information. Overexpenditure of budgeted 
resources could be fatal to a ^development project, especially if the 
quality of the product will suffer as a consequence. 

4# Simple Observation# Informal observations made by the formative evalu- 
ator should provide substantial pay-off# Conversation sampling of 
teacher or student discussions about early developmental efforts or of 
project staff members during and after work will undoubtedly produce 
useful judgment data# This is a powerful tool of the formative evalu- 
ator# 

One final category of interim formative evaluation activity is critical for 
large-scale development projects# This one deals with management, with evaluating 
the process of developing the product rather than the product itself# Such an eval- 
uation will, of course, require different sorts of information than will the evalu- 
ation of the product. Stuff lebeam (1968, 1968, 1971) has elaborated to procedures 
for monitoring programs as **process evaluation" in the CIPP model# By the evalua- 
tion of program operations, we are referring to something different from process 
evaluation; we are referring to an evaluation of the operational design of the 
project development. Criteria will be set up, the operational plan will be scruti- 
nized and judgments will be made about the operations# For large, complex projects, 
operations analysis techniques would be useful tools for the formative evaluator. 
Management tools such as PERT and PPBS are also relevant techniques for the interim 

formation evaluator * 

Normative Product Evaluation . 

The fourth type of activity in which formative evaluators engage is that 
of formative product evaluation# This category of formative evaluation activity 
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has most often been associated with product validation studies or feasibility 
studies* We think there are other evaluative activities that fall within this 
category as well* For example; descriptive analyses of the type described under 
interim formative evaluation activities (now applied to the entire product) and 
content analysis techniques are extremely important at this point* 

By formative product evaluation, we don't mean a summative or consumer 
reports-type of appraisal. We also don't mean the evaluation of pieces of the 
final product* Instead, we are suggesting that an important formative evaluation 
activity is the evaluation of the product as it has been put together strictly for 
feedback to the developer * Anderson (1969) provided an excellent example of what 
he called a "field test"* We would label such a study as a formative product evalu- 
ation study* Borich (1971) suggested a conceptual model for formative product eval- 
uation* Validation of a product with a s mple of subjects from the target popula- 
tion or a feasibility study of a plan for educational change are the most frequently 
found formative product evaluation studies in the literature* 

Knowledge about the extent to which valued objectives are achieved with a 
plan or product are important, but should not be the only formative product con- 
sideration* Cost analyses should be another formative product evaluation concern 
in the appraisal of a first draft completion of educational product or plan* 
Wilkinson (1971) described the following kinds of cost analysis that might be built 
into a formative product evaluation plan: 

1* Traditional budgeting - establishment of expense categories, appro- 
priation of resources to organizational units, and authorization of ex- 
penditures by organizational units* 

2* Cost accounting - measurement of output quality and quantity at desig- 
nated points in the educational process and assignment of costs in- 
curred to achieve that output* 
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3. Cost analysis - definition of objectives and strategies to meet 
objectives; determination of resources required to mount strategies and 
the conversion of resources to dollar amounts to aid the establishment 
of budgets and information systems* 

4. Cost effectiveness analysis - comparison of estimated resources required 
for alternative strategies to meet objectives with estimated output of 
the strategies to aid in the selection of a strategy* 

5. Cost-benefit analysis - comparison of all relevant resources required 
to achieve an objective with the likely benefits (dollar value of 
results) to aid in making decisions about the desirability of initiating 
revising, continuing or terminating a program* 

Scriven (1971) coined the term, goal-free evaluation, to refer to evaluation 
completely divorced from goal statements* He defined the role of the goal -free 

formative evaluator as follows: 

The GFE can work in the formative role * * . but he will not 
be doing what most formative evaluators do* He will not be spending 
much of his time helping the project staff convert their plans into 
behavioral objectives. He will not be advising them on probable 
mismatch between their abstract goals and some Implicit commitments 
of their materials. He will not be constructing biserial correlation 
matrices. for item analysis on their quizzes* He will simply get a 
look at the materials plus procedures ( or descriptions of what 
fchay will bo like) and a deadline by which time he must get evalu- 
ative feedback into the rewrite/replan process* (Scriven, 1971, p* Bz) 

The GFE will inspect the domain of possible effects vary carefully, looking for 

signs of good and bad aspects of the product under scrutiny* We suggest that a 

goal-free evaluation of the. product as suggested by Cronbach (1963), Stake (1967), 

Scriven (1967, 1971), and Walberg (1970) will provide information about the 

strengths and weaknesses of the product being appraised* Walburg suggested that 

one might look for effects in areas identified by: 

1* existing goal statements 

2* analysis of materials 
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3 . the domain covered by a battery of Items In the general content 
area (see , for example, Metfessei and Michael, 1969)* 

Scrlven warned the goal-free evaluator to avoid looking at (being Influenced by) 
goal statements until after he has Inspected the ground covered by the materials he 
is evaluating. Walberg's second and third sources of information about effects 
are noteworthy, however. In addition, the goal-free evaluator will want to obtain 
advice from content and media experts on what effects to look for. 

The sources of external and internal information listed under formative in- 
terim evaluation activities are applicable to formative product evaluation 
activities also. The object under scrutiny at this point will be the entire 
assembled product, however, rather than its components. Contextual information is 
of utmost importance at this point. The formative product evaluation should test 
frh ft product in the context within which it is Intended to function. If the 
defined context is very broad, so too should be the contexts investigated in the 
field test. It is at this stage that a relationship can be established between the 
Intended context, internal characteristics, and external characteristics of the 
product. In this way the developer, when confronted with the need to revise, has 
a better chance of determining the revision most likely to improve the product. 

An illustration might help to clarify this point. Anderson (1969), in his 
formative product evaluation, collected information of all three types (although 
his emphasis was undoubtedly on external information). When he administered the 
programmed text he was evaluating two schools, he found a rather sharp difference 
between the schools as to the programme effectiveness. The formative evaluator 
often will not be satisfied with observing this outcome; he might need to find 
out why it occurred If he is to have complete information concerning possible re- 
visions of the materials. After finding the school differences, Anderson (1969) 
began to search for possible causes of the descrepancy between schools in a m a nner 
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very similar to that employed by the educational researcher* He hypothesized 
various causes , then systematically (to the extent jjossible) investigated those hy- 
potheses* Fortunately he had collected attitudnal and utilization data from 
both students and teachers* He found that teachers in the poorer achieving school 
did not require students to read the program while teachers in the other school did. 

More students reported not finishing the program in the poorer achieving school than 
the other, a rather important bit of information when evaluating a product* s 
effectiveness. It can be seen that certain assumptions that Anderson (1969) made 
about the context within which the product would be used were not met in one school* 

He finally had a choice of making the kind of context necessary foi the product to 
work more explicit or revising his intended contexts and changing the product to 
match those contexts. The important point to this paper is that without the 
supporting contextual information, Anderson (1969) would have been unable to account 
for his results and, thus would have been less effective in aiding sound product 
development* Another point to note, however, is that explanatory information is 
not always needed for formative product evaluation work, but it can be critical and 
.should not be overlooked. - M ' 



DISCUSSION 

We have found the framework just described to be useful for considering the 
functions of formative evaluators (see, for example, Sanders, 1971)* We have been 
able to categorize formative evaluation activities with very little difficulty 
(perhaps a good indication that this is a useful framework). We don*t suggest that 
all formative evaluators in product development projects should attempt to follow 
the structure in a step-wise fashion (i.e. , this is not to be construed as an 
exhaustive, linear model). Rather, we recommend that formative evaluators and 
developers use the franework to describe the evaluative activities in which they 
are engaged* Further, the framework can ba used to identify the types of formative 
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evaluation activity in which project personnel are engaged at any one point in time, 
to identify and organize "next steps" in formative evaluation, and to identify (and 
perhaps accomodate) missing elements in a formative evaluation design. To have one 
person conduct a needs assessment, evaluate objectives, conduct a series of interim 
evaluation activities, and then produce several formative product evaluation 
studies would take a great amount of resources and that sequence of activity may 
not be appropriate. Available information, such as published needs assessment 
studies for use in developing a cogent rationale for a sat of objectives, should be 
used whenever possible. 

The summary of techniques contained in Figure 3 suggests definite directions 
for trainers of formative evaluators to take. It seems clear that formative evalu- 
ation is not a unidisciplinary undertaking and that inquiry methods currently used 
in a wide range of disciplines need to be applied, when relevant, in answering 
formative evaluation questions. The toolkit of the formative evaluation should be '' 

i 

greatly expanded and we propose that the logical analysis summarized in Figure 3 
may be a first step toward listing relevant tools. The reader should be able to 
identify a close relationship between the framework presented herein and extent ^ 

developmental models since the processes of development and formative evaluation 
are interdependent. Designing formative evaluation activities around the conceptual 
framework presented herein should alleviate many anxieties of developers about 
potential surprise criticisms in a summative evaluation. Thus, we recommend that 
the activities described in this paper become an integral part of any systematic 
development effort to the extend that project resources or contraints allow. 




23 



Figure 1 



A Structure for Planning Formative Evaluation in Product Development 



1. Pre -Developmental Activities 
A* Needs Assessment 1 
B. Evaluation of Needs 
11. Evaluation of Objectives 

A. logical Analyses 

1. Cogency of Rationales for Objectives 

2. Consequences of Reaching Objectives 
3* Appeals to Higher Values 

B. Empirical Analyses 

1* Evaluation by Relevant Groups 

2. Evaluation by Specialists 

3. Appeals to Written Documents 
HI. Formative Interim Evaluation 

A. Formal 

1* Pay-off Evaluation 

2. Intrinsic Evaluation 

3. Evaluation of Program operations 



B. Informal (Unobtrusive) 
XV. Formative Product Evaluation 

A. Validation Studies 

B. Cost Analyses 

C* Descriptive Analyses 
D. Goal Free Evaluation 



Although such an activity is not evaluative as it stands by Itself, it is 
included in this framework as a pre-evaluation or evaluation-supporting activity. 




Figure 2 

A Classification of Information Needs in Formative Evaluation 

I. Internal Information 

A* Descriptive Information 

1. Physical specifications 

2. Rationale, goals, and objectives 

3. Content 

4. Pedagogical strategy 

5. Other 

B. Critical Appraisal 

1. Author (developer) 

2. Experts (subject matter, media, psychologists, etc.) 

3. Students using the materials 

4. Teachers using the materials 

5. Relevant others 
XI. External Information 

A. Assessment of the effects of the materials on student behavior 

1. Achievement 

2. Attitude 
3* Skill 

4. Interest 

5 . Commitment 

6. Other 

B. Assessment of the effects of the material* on teacher behavior 

1. Attitude 

2. Interest 
3« Comnitment 
4. Competency 
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Figure 2 (cont) 



5. Teaching strategy 
6* Other 

C. Assessment of the effects of the materials on the behavior of relevant 
others* 

1* Parents 

2* Administrators 

3* Teachers not using the materials 
4. Students not using the materials 
3* The community 
6* Other 

III* Contextural Information 

A* Student Characteristics 
3* Teacher characteristics 
C* School characteristics 

D. Community characteristics 
B* Curricular characteristics 

F* Other relevant elements in the learning environment 




SOURCE OF INFORMATION 



Figure 3 



Summary of Techniques and Procedures 
Appropriate for Formative Evaluation 



ERiC 



formative evaluation activity 




evaluation of 




PRODUCT 


PRE -DEVELOPMENTAL 


objectives 


INTERIM 


logical analyses 


logical analyses: 


materials analysis 


cost analyses 


of needs: 


1. cogency 


guidelines 


materials analysis 


1. cogency 


2. consequences 




2. consequences 


3. higher order 


content analysis 


guidelines 


3. higher order 


values 






values 






content analysis 


empirical analyses: 


empirical analyses: 


analysis of learn- 


of needs: 

1* group data: 


1, group data: 


ing structures 


group data £riti- 


surveys 


surveys 


group data Criti- 


scaling 


scaling 


cal appraisal) 


cal appraisal) 


Q- technique 


Q- technique 


expert opinion 


expert opinion 


semantic dif- 


semantic dif- 


ferential 


ferential 


(including audios) 




Delphi tech- 


Delphi tech- 




unobtrusive mea- 


nique 


nique 


unobtrusive mea- 


sentence com- 


sentence com- 


sures 


sures 


pletion 


pletion 






2. observation & 


2» observation & 






expert opinion 


expert opinion 






unobtrusive 


unobtrusive 


PERT 




measures 


measures 


PPBS 




accrediation 


accrediation 


system analysis 




procs . 


procs. 






category sys- 


category sys- 






tems 


tems 






rating systems 


rating systems 






3* analysis of 


3. analysis of 






documents 


documents 






unobtrusive 


unobtrusive 






measures 


measures 






content ana- 


content ana- 






lysis 


lysis 








operationalization 


experimental and 


experimental and 




of objectives 


quasi -experimental 


quasi -experimental 




design 


design; hypothesis 




experimental try- 
out of goal state- 


clinical methods 


testing 




ments 


quantitative natu- 


cost analyses 






ralistic observa- 
tion techniques 


6FE 






correlational ana- 






unobtrusive mea- 


lyses 






sures 


quantitative natu- 
ralistic observa- 
tion techniques 














CONTEXTUAL 



Figure 3 
(cont) 



PRE- DEVELOPMENTAL 


EVALUATION OF 
OBJECTIVES 


INTERIM 


PRODUCT 








unobtrusive measure 








group data per- 
ceived £n effec- 
tiveness of product) 








observation tech- 
nique 


needs assessment 


context assessment 
(if no needs 
assessment results 
available) 


literature re- 
views 

informal obser- 
vation 


ATI procedures 

context assessment 
(focus on external 
validity) 
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